<html>
<head><meta charset="utf-8"><title>design meeting 2020.02.07 · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html">design meeting 2020.02.07</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="187638839"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187638839" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187638839">(Feb 07 2020 at 13:31)</a>:</h4>
<p>Hey <span class="user-group-mention" data-user-group-id="897">@T-compiler/meeting</span> -- reminder that we have a <strong>design meeting</strong> today in 1.5 hours. We'll be discussing <a href="https://github.com/rust-lang/compiler-team/issues/237" target="_blank" title="https://github.com/rust-lang/compiler-team/issues/237">compiler-team#237</a> ("parser librar-yification"). From the issue:</p>
<blockquote>
<p>The goal of the meeting is twofold:</p>
<ul>
<li>
<p>share general experience with IDE-flavored parsing/trees in rust-analzyer (<a href="https://hackmd.io/XoQrzR8GRLa64jpjylQ7Bw" target="_blank" title="https://hackmd.io/XoQrzR8GRLa64jpjylQ7Bw">writeup</a>)<br>
 *get feedback on a semi concrete plan for the first steps of parser library-ification (<a href="https://hackmd.io/ifjST_Y4R-SQ0AWiEcU6FQ" target="_blank" title="https://hackmd.io/ifjST_Y4R-SQ0AWiEcU6FQ">writeup</a>)</p>
</li>
<li>
<p>form consensus on what the next steps should be (and if we need this work in the first place)</p>
</li>
</ul>
</blockquote>
<p>In particular, the hope is that we can discuss some of the complications that would arise and how they might be handled.</p>



<a name="187646129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646129">(Feb 07 2020 at 15:01)</a>:</h4>
<p>Hello <span class="user-group-mention" data-user-group-id="897">@T-compiler/meeting</span> , shall we start? Please say <span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span> if you are here :-)</p>



<a name="187646256"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646256" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646256">(Feb 07 2020 at 15:02)</a>:</h4>
<p>also, cc <span class="user-group-mention" data-user-group-id="1060">@WG-rls2.0</span></p>



<a name="187646392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646392">(Feb 07 2020 at 15:04)</a>:</h4>
<p>Let's maybe start with discussing what we have in rust-analyzer?</p>
<p>Did people have a chance to read the write up? </p>
<p>Are there any specific questions about that?</p>



<a name="187646574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646574">(Feb 07 2020 at 15:06)</a>:</h4>
<p>(Hey, sorry, was running a few minute late, thanks for picking up the ball <span class="user-mention" data-user-id="133169">@matklad</span> =)</p>



<a name="187646608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646608">(Feb 07 2020 at 15:06)</a>:</h4>
<p>I read the write-up; I have one or two questions</p>



<a name="187646623"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646623" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646623">(Feb 07 2020 at 15:06)</a>:</h4>
<p>Some thoughts while reading the write-ups (adding more over time...):</p>
<ul>
<li>The general notes about optimizations of the tree weren't all that important for structuring the parser. I think we should mostly focus on the interface the parser uses.</li>
<li>Unless there are strong reason to go with <code>token![]</code> I would like to retain the current <code>token::Colon</code>, etc.</li>
</ul>



<a name="187646624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646624">(Feb 07 2020 at 15:06)</a>:</h4>
<p>First off, which parts of rust-analyzer work directly from the underlying, untagged "green" syntax trees?</p>



<a name="187646672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646672">(Feb 07 2020 at 15:07)</a>:</h4>
<blockquote>
<p>To deal with precedence in cases like $expr * 1, we use special invisible parenthesis, which are explicitelly handled by the parser</p>
</blockquote>
<p>I believe that's the case today.</p>



<a name="187646785"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646785" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646785">(Feb 07 2020 at 15:08)</a>:</h4>
<blockquote>
<p>Parsing is resilient (even if the input is invalid, parser tries to see as much syntax tree fragments in the input as it can).</p>
</blockquote>
<p>It seems like the write-up is claiming that the current parser doesn't do this, but I don't think that's the case. The parser does a whole lot of recovery.</p>



<a name="187646803"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646803" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646803">(Feb 07 2020 at 15:08)</a>:</h4>
<blockquote>
<p>which parts of rust-analyzer work directly from the underlying, untagged "green" syntax trees?</p>
</blockquote>
<p>So, green and untagged are not the same</p>



<a name="187646811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646811">(Feb 07 2020 at 15:08)</a>:</h4>
<p>green trees are untagged</p>



<a name="187646851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646851">(Feb 07 2020 at 15:09)</a>:</h4>
<p>however, there's also <code>SyntaxNode</code> layer above green trees, whcih is also untagged (but, unlike G, has parent pointers)</p>



<a name="187646931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646931">(Feb 07 2020 at 15:10)</a>:</h4>
<p>It's more or less true that nothing in rust-analyzer works directly with green trees (the only thing that touches green trees is the code that builds syntax tree in the first place)</p>



<a name="187646962"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187646962" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187646962">(Feb 07 2020 at 15:10)</a>:</h4>
<p><em>Some</em> code in rust-analyzer works with untagged representation with parent pointers. That is mostly the code close to the IDE</p>



<a name="187647007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647007">(Feb 07 2020 at 15:11)</a>:</h4>
<p>For example, "flip two things around the <code>,</code>" assist is implemented in terms of untagged SyntaxNodes</p>



<a name="187647021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647021">(Feb 07 2020 at 15:11)</a>:</h4>
<p>Yes, OK, that makes sense.</p>



<a name="187647059"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647059" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647059">(Feb 07 2020 at 15:11)</a>:</h4>
<p>As well as a pretty important layer which takes a raw offset into a file and figures out what semantic element exists at that position</p>



<a name="187647150"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647150" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647150">(Feb 07 2020 at 15:12)</a>:</h4>
<p>That code climbs untagged tree up from the cursor position to find an item, and than finds a semantic repr for that item</p>



<a name="187647178"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647178" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647178">(Feb 07 2020 at 15:12)</a>:</h4>
<p>Other notes:</p>
<blockquote>
<p>If possible, errors are not reported during parsing and are postponed for a separate validation step. For example, parser accepts visibility modifiers on trait methods, but then a separate tree traversal flags all such visibilites as erroneous.</p>
</blockquote>
<ul>
<li>
<p>It's important that no new syntax be accepted <em>pre-expansion</em> unless with the express approval of the language team. Could you elaborate on "separate validation step"... does it run before or after expansion?</p>
</li>
<li>
<p>Similarly, the current parser needs to collect a set of spans for pre-expansion gating. How would we deal with that in the new model?</p>
</li>
</ul>



<a name="187647411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647411">(Feb 07 2020 at 15:14)</a>:</h4>
<blockquote>
<p>does it run before or after expansion?</p>
</blockquote>
<p>Before expansion. From the point of view of highler-level layers, it is indivisible from the <code>parse</code> phase. That is, it is an impl detail of parsing</p>



<a name="187647430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647430">(Feb 07 2020 at 15:14)</a>:</h4>
<p>Regarding:</p>
<div class="codehilite"><pre><span></span><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">TokenSource</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">current</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Token</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">lookahead_nth</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">n</span>: <span class="kt">usize</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Token</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">is_keyword</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kw</span>: <span class="kp">&amp;</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="kt">bool</span><span class="p">;</span><span class="w"></span>

<span class="w">    </span><span class="k">fn</span> <span class="nf">bump</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<ul>
<li>I assume we can (and <em>will</em>) encode higher level operations on this such as e.g. <code>eat_keyword(..)</code> and whatnot.</li>
<li>Can we replace <code>kw: &amp;str</code> with <code>kw: Symbol</code>?</li>
</ul>



<a name="187647462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647462">(Feb 07 2020 at 15:14)</a>:</h4>
<blockquote>
<p>How would we deal with that in the new model?</p>
</blockquote>
<p>I think this also belongs to the validation phase</p>



<a name="187647567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647567">(Feb 07 2020 at 15:15)</a>:</h4>
<p>I'm feeling a bit lost, I have to admit, but I do think it's good to turn towards the parser integration</p>



<a name="187647679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647679">(Feb 07 2020 at 15:16)</a>:</h4>
<p>/me pulls up the document</p>



<a name="187647749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647749">(Feb 07 2020 at 15:17)</a>:</h4>
<p>I guess I'd love it if someone (<span class="user-mention" data-user-id="126931">@centril</span>?) wanted to try and compare/contrast the "current" and "new" model</p>



<a name="187647759"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647759" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647759">(Feb 07 2020 at 15:17)</a>:</h4>
<blockquote>
<p>I assume we can (and will) encode higher level operations on this such as e.g. eat_keyword(..) and whatnot.</p>
</blockquote>
<p>Correct, but not as a part of this interface. Rather, that would be utility methods on the parser.</p>
<blockquote>
<p>can we replace kw: &amp;str with kw: Symbol?</p>
</blockquote>
<p>I'd rather not to: if we do, we'd have to make the set of keywords a part of the interface. But maybe I am not exactly understanding what is Symbol</p>



<a name="187647868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647868">(Feb 07 2020 at 15:18)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> it's an interned string representation with some things declared upfront, <a href="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/symbol/struct.Symbol.html" target="_blank" title="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/symbol/struct.Symbol.html">https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/symbol/struct.Symbol.html</a></p>



<a name="187647872"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647872" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647872">(Feb 07 2020 at 15:18)</a>:</h4>
<p>(I also think details about the specifics of the API seem a bit "lower level", unless they map to some higher-level distinctions -- e.g., I could see that <code>&amp;str</code> vs <code>Symbol</code> represents something important)</p>



<a name="187647923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187647923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187647923">(Feb 07 2020 at 15:18)</a>:</h4>
<blockquote>
<p>Correct, but not as a part of this interface. Rather, that would be utility methods on the parser.</p>
</blockquote>
<p>That's all good.</p>



<a name="187648004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648004">(Feb 07 2020 at 15:19)</a>:</h4>
<p>Agree that we should probably aim at teasing apart the high-level constraints here.</p>
<p>In particular, one constraint I have in mind is that the parser should not care about interning</p>



<a name="187648007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648007">(Feb 07 2020 at 15:19)</a>:</h4>
<p>Maybe let's start with how existing parser and expansion intertwine? e.g., <span class="user-mention" data-user-id="126931">@centril</span> noted that the precise set of things parser accepts "pre-expansion" is part of the language definition, maybe we can sketch that out a bit more</p>



<a name="187648010"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648010" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648010">(Feb 07 2020 at 15:19)</a>:</h4>
<p>I would be happy with e.g. <code>const Const: &amp;str = "const"</code> instead</p>



<a name="187648098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648098">(Feb 07 2020 at 15:20)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> so <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> is the resident expansion expert</p>



<a name="187648102"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648102" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648102">(Feb 07 2020 at 15:20)</a>:</h4>
<p>I think if we had a kind of "idealized" model of how parser-expansion etc are interacting, it might be clearer how to express matklad's proposal as a diff on that</p>



<a name="187648138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648138">(Feb 07 2020 at 15:20)</a>:</h4>
<p>(sorry, didn't have time to do a write-up to compare, had the fever yesterday &amp; today)</p>



<a name="187648144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648144">(Feb 07 2020 at 15:20)</a>:</h4>
<p>No doubt, but they're not present, still I don't think we need the details of name resolution at the moment</p>



<a name="187648196"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648196" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648196">(Feb 07 2020 at 15:21)</a>:</h4>
<p>I could try to skech a view and you can correct it :)</p>



<a name="187648223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648223">(Feb 07 2020 at 15:21)</a>:</h4>
<p>So to give a quick overview of pre-expansion gating...</p>



<a name="187648235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648235">(Feb 07 2020 at 15:21)</a>:</h4>
<p>from what I undestand, the biggest diff would be that the current model very much builds on top of specific AST</p>



<a name="187648251"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648251" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648251">(Feb 07 2020 at 15:21)</a>:</h4>
<p>(I am taking notes in <a href="https://hackmd.io/MR4bSmcpSlOPhz6aSebpXQ" target="_blank" title="https://hackmd.io/MR4bSmcpSlOPhz6aSebpXQ">this hackmd</a>, fyi)</p>



<a name="187648270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648270">(Feb 07 2020 at 15:22)</a>:</h4>
<p>The parser collects a set of spans into a sink: <a href="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/parse/struct.GatedSpans.html" target="_blank" title="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/parse/struct.GatedSpans.html">https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/parse/struct.GatedSpans.html</a></p>



<a name="187648323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648323">(Feb 07 2020 at 15:22)</a>:</h4>
<p>mainly using <a href="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/parse/struct.GatedSpans.html#method.gate" target="_blank" title="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/parse/struct.GatedSpans.html#method.gate">https://doc.rust-lang.org/nightly/nightly-rustc/rustc_session/parse/struct.GatedSpans.html#method.gate</a></p>



<a name="187648384"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648384" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648384">(Feb 07 2020 at 15:22)</a>:</h4>
<p>Let's back up one more step :)</p>



<a name="187648397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648397">(Feb 07 2020 at 15:23)</a>:</h4>
<p>The ida here is that we introduce feature-gated syntax</p>



<a name="187648407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648407">(Feb 07 2020 at 15:23)</a>:</h4>
<p>when dealing with macros, which call into the black-box parsers, the sink is snapshotted, emptied, and then on a successful macro arm the gated spans are merged</p>



<a name="187648413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648413">(Feb 07 2020 at 15:23)</a>:</h4>
<p>e.g., <code>async fn</code> (at one point)</p>



<a name="187648429"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648429" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648429">(Feb 07 2020 at 15:23)</a>:</h4>
<p>and the parser accepts it, but records the span where it occurred</p>



<a name="187648435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648435">(Feb 07 2020 at 15:23)</a>:</h4>
<p>or take <code>|</code> or-patterns today</p>



<a name="187648449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648449">(Feb 07 2020 at 15:23)</a>:</h4>
<p>then -- after parsing -- we check the set of feature gates to make sure that the syntax was enabled</p>



<a name="187648523"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648523" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648523">(Feb 07 2020 at 15:24)</a>:</h4>
<p>we do this in part because macro-rules has things like <code>$e:expr</code> -- and the idea is that <code>$e</code> if matches this syntax without the feature gate, but we then remove that syntactic form, then we have changed the behavior of stable programs</p>



<a name="187648526"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648526" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648526">(Feb 07 2020 at 15:24)</a>:</h4>
<p>(correct?)</p>



<a name="187648543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648543">(Feb 07 2020 at 15:24)</a>:</h4>
<p>yep</p>



<a name="187648558"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648558" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648558">(Feb 07 2020 at 15:24)</a>:</h4>
<p>(here's the or-pattern example, <a href="https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_parse/parser/pat.rs.html#120" target="_blank" title="https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_parse/parser/pat.rs.html#120">https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_parse/parser/pat.rs.html#120</a> )</p>



<a name="187648616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648616">(Feb 07 2020 at 15:25)</a>:</h4>
<p>note that fragments feature gated may not be manifest in the AST itself</p>



<a name="187648620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648620">(Feb 07 2020 at 15:25)</a>:</h4>
<p>One implication -- and this is somewhat contentious -- of the current setup is that you can only use <code>cfg</code> to include "unstable" syntax by embedding that syntax into a separate file</p>



<a name="187648622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648622">(Feb 07 2020 at 15:25)</a>:</h4>
<p>Is collection of spans more powerful than  building a concrete syntax tree, and then sanitizing the tree?</p>



<a name="187648647"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648647" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648647">(Feb 07 2020 at 15:25)</a>:</h4>
<p>e.g., <code>#[cfg(false)] mod foo;</code> works no matter that <code>foo.rs</code> contains, because it never parses <code>foo.rs</code> -- but maybe it's worth talking about the flow of how that comes about?</p>



<a name="187648664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648664">(Feb 07 2020 at 15:25)</a>:</h4>
<p>it implies that the parsing of files is entwined with cfg expansion</p>



<a name="187648679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648679">(Feb 07 2020 at 15:26)</a>:</h4>
<p>For example <code>let | p = 0;</code> gates <code>|</code> despite not being in the AST</p>



<a name="187648734"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648734" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648734">(Feb 07 2020 at 15:26)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> that's <a href="https://github.com/rust-lang/rust/issues/64197" target="_blank" title="https://github.com/rust-lang/rust/issues/64197">https://github.com/rust-lang/rust/issues/64197</a>, a bug that should be fixed in either case</p>



<a name="187648767"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648767" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648767">(Feb 07 2020 at 15:27)</a>:</h4>
<blockquote>
<p>Is collection of spans more powerful than  building a concrete syntax tree, and then sanitizing the tree?</p>
</blockquote>
<p>this is a good question, <span class="user-mention" data-user-id="133169">@matklad</span>, I don't think so</p>



<a name="187648808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648808">(Feb 07 2020 at 15:27)</a>:</h4>
<p>it is if the CST does not have the thing gated manifest in the CST</p>



<a name="187648823"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648823" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648823">(Feb 07 2020 at 15:27)</a>:</h4>
<p>So, <span class="user-mention" data-user-id="126931">@centril</span>, the point of <a href="https://github.com/rust-lang/rust/issues/64197" target="_blank" title="https://github.com/rust-lang/rust/issues/64197">#64197</a> is that presently "cfg expansion" is duplicated between parser and expansion?</p>



<a name="187648923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648923">(Feb 07 2020 at 15:28)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> what happens is that the parser drives conditional compilation when it comes to modules</p>



<a name="187648931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648931">(Feb 07 2020 at 15:28)</a>:</h4>
<blockquote>
<p>it is if the CST does not have the thing gated manifest in the CST</p>
</blockquote>
<p>where CST is "concrete syntax tree" ? But shouldn't that by definition ...</p>



<a name="187648937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648937">(Feb 07 2020 at 15:28)</a>:</h4>
<p>and <code>config.rs</code> lives in the parser</p>



<a name="187648939"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648939" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648939">(Feb 07 2020 at 15:28)</a>:</h4>
<p>i.e., the parser has a loop that is kind of like</p>
<ul>
<li>parse module file <code>foo.rs</code></li>
<li>identify <code>mod bar;</code> declarations in there and determine if they are cfg'd out</li>
<li>if not, then parse them</li>
</ul>



<a name="187648948"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648948" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648948">(Feb 07 2020 at 15:28)</a>:</h4>
<p>(did you write CST in one instance where you meant to write AST?)</p>



<a name="187648993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187648993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187648993">(Feb 07 2020 at 15:29)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> no; CST is somewhat ambiguous</p>



<a name="187649002"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649002" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649002">(Feb 07 2020 at 15:29)</a>:</h4>
<p>("the real AST is HIR")</p>



<a name="187649020"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649020" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649020">(Feb 07 2020 at 15:29)</a>:</h4>
<p>or are you referring indeed to cases like <code>mod bar;</code> where the concrete syntax itself does not have the contents of <a href="http://bar.rs" target="_blank" title="http://bar.rs">bar.rs</a> ?</p>



<a name="187649047"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649047" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649047">(Feb 07 2020 at 15:29)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> I'm referring to whether the leading  <code>|</code> in <code>let | p = 0;</code> is in the tree or not</p>



<a name="187649054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649054">(Feb 07 2020 at 15:30)</a>:</h4>
<p>it's not in the current "AST"</p>



<a name="187649106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649106">(Feb 07 2020 at 15:30)</a>:</h4>
<p>before we go too deep, can we confirm that <a href="#narrow/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07/near/187648939" title="#narrow/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07/near/187648939">my summary here</a> is correct?</p>



<a name="187649129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649129">(Feb 07 2020 at 15:30)</a>:</h4>
<p>(after that, I think it'd clearly be good to define our terms like CST/AST/HIR and make sure we're all using them to refer to the same things)</p>



<a name="187649132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649132">(Feb 07 2020 at 15:30)</a>:</h4>
<p>Okay.  I am under impression that in matklad's proposal, it might be included (if only as part of "trivia" ?)</p>



<a name="187649165"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649165" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649165">(Feb 07 2020 at 15:30)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> high level idea is correct, but I think you're getting into the weeds</p>



<a name="187649188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649188">(Feb 07 2020 at 15:31)</a>:</h4>
<p>I'm sure both old and new parser can deal with it</p>



<a name="187649211"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649211" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649211">(Feb 07 2020 at 15:31)</a>:</h4>
<p>Correct, I specifically use CST as a tree that, when printed, is guaranteed to yield exactly the source text. So which contains all tokens, and all trivia</p>



<a name="187649221"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649221" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649221">(Feb 07 2020 at 15:31)</a>:</h4>
<p>More specifically, the parser would do less, and leave things to conditional compilation later</p>



<a name="187649333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649333">(Feb 07 2020 at 15:32)</a>:</h4>
<p>The model we hope to move towards is one where that loop moves to the expander</p>



<a name="187649350"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649350" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649350">(Feb 07 2020 at 15:32)</a>:</h4>
<p>and the parser itself is basically just a function that parses a single file</p>



<a name="187649353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649353">(Feb 07 2020 at 15:32)</a>:</h4>
<p>(Confirm/deny?)</p>



<a name="187649356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649356">(Feb 07 2020 at 15:33)</a>:</h4>
<p>confirm</p>



<a name="187649360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649360">(Feb 07 2020 at 15:33)</a>:</h4>
<p>Confirm</p>



<a name="187649372"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649372" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649372">(Feb 07 2020 at 15:33)</a>:</h4>
<p>OK <span aria-label="+1" class="emoji emoji-1f44d" role="img" title="+1">:+1:</span></p>



<a name="187649383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649383">(Feb 07 2020 at 15:33)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> I don't think there's any disagreement here btw; it seems everyone agrees that this is a bug that should be fixed</p>



<a name="187649391"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649391" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649391">(Feb 07 2020 at 15:33)</a>:</h4>
<p>only engineering hours has prevented fixing it</p>



<a name="187649411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649411">(Feb 07 2020 at 15:33)</a>:</h4>
<p>The reason I'm kind of going through this is</p>



<a name="187649466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649466">(Feb 07 2020 at 15:34)</a>:</h4>
<p>I want to kind of create a "high level picture" of how this process works, and clearly identify which bits are the responsibility of which</p>



<a name="187649501"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649501" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649501">(Feb 07 2020 at 15:34)</a>:</h4>
<p>Anyway, that's as deep as I want to go on that topic, but I wanted to try and add now the pre-expansion gating in this picture a bit ...</p>



<a name="187649533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649533">(Feb 07 2020 at 15:35)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> but assuming we store details like leading <code>|</code> then I agree in principle there's no problem with preexp gating</p>



<a name="187649715"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649715" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649715">(Feb 07 2020 at 15:37)</a>:</h4>
<p>I have some more questions about pre-expansion gating</p>



<a name="187649724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649724">(Feb 07 2020 at 15:37)</a>:</h4>
<p>It seems relevant, but I'm also happy to take some notes and pursue them later</p>



<a name="187649729"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649729" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649729">(Feb 07 2020 at 15:37)</a>:</h4>
<p>I guess, on the highest-possible level the current state is (module <code>mod foo;</code> bug fixed) is this:</p>
<ul>
<li>parser takes as an input the stream of spanned tokens and some state</li>
<li>the parser produces an AST data structure, and record auxilary information into the state along the way (like the gated spans)</li>
</ul>



<a name="187649759"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649759" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649759">(Feb 07 2020 at 15:37)</a>:</h4>
<p>(One thing I'm curious to dig a bit more into is recovery and prediction, for example)</p>



<a name="187649831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649831">(Feb 07 2020 at 15:38)</a>:</h4>
<blockquote>
<p>(One thing I'm curious to dig a bit more into is recovery and prediction, for example)</p>
</blockquote>
<p>I acutaly think that this is a well understood and boring bit. Like, you just write that code, it works.</p>



<a name="187649834"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649834" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649834">(Feb 07 2020 at 15:38)</a>:</h4>
<p>I was going to write some bullets very much like that <span class="user-mention" data-user-id="133169">@matklad</span>; I think what you're proposing could be seen as splitting that "parser: tokens -&gt; ast" interace so that there are more intermediate steps?</p>



<a name="187649868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649868">(Feb 07 2020 at 15:38)</a>:</h4>
<p>Oh, well, depending on which part of the recovery you are talking about</p>



<a name="187649880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649880">(Feb 07 2020 at 15:38)</a>:</h4>
<p>tweaking parser to recover is easy</p>



<a name="187649886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649886">(Feb 07 2020 at 15:38)</a>:</h4>
<p>My understanding is that we move to something more like syn, except for spans, we don't use spans</p>



<a name="187649918"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649918" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649918">(Feb 07 2020 at 15:39)</a>:</h4>
<blockquote>
<p>tweaking parser to recover is easy</p>
</blockquote>
<p>And to be clear, I think the current parser does  a shit tonne of this</p>



<a name="187649922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187649922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187649922">(Feb 07 2020 at 15:39)</a>:</h4>
<p>tweaking syntax trees to incorporate incomplete syntax is hard, and is, in some sence, the core of teh proposal</p>



<a name="187650060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650060">(Feb 07 2020 at 15:40)</a>:</h4>
<blockquote>
<p>And to be clear, I think the current parser does a shit tonne of this</p>
</blockquote>
<p>I haven't looked into this recently, but the last time I did (around original libsytnax2 RFC) the recovery wasn't quite the one needed for IDEs</p>



<a name="187650071"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650071" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650071">(Feb 07 2020 at 15:40)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> speaking of recovery; I think it's important that we be able to retain e.g. higher level combinators like <a href="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html#method.parse_delim_comma_seq" target="_blank" title="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html#method.parse_delim_comma_seq">https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html#method.parse_delim_comma_seq</a></p>



<a name="187650076"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650076" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650076">(Feb 07 2020 at 15:40)</a>:</h4>
<p>But I do belive that these are detains</p>



<a name="187650134"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650134" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650134">(Feb 07 2020 at 15:41)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> but could you briefly elaborate on what sort of recovery you think was missing?</p>



<a name="187650148"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650148" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650148">(Feb 07 2020 at 15:41)</a>:</h4>
<p>(I think most of the current parser code is actually recovery...)</p>



<a name="187650254"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650254" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650254">(Feb 07 2020 at 15:42)</a>:</h4>
<p>can we come back to this core parser interface? it seems like the details kind of matter.</p>
<p><span class="user-mention" data-user-id="133169">@matklad</span> said earlier (I believe) that the current model for the parser is that it takes as input:</p>
<ul>
<li>the same tokens proc-macro specifies (which are fairly open-ended)</li>
</ul>
<p>and produces  as output</p>
<ul>
<li>an AST plus some auxiliary stuff (e.g., information about where unstable syntax was used in parsing that AST).</li>
</ul>
<p>I think that the question is whether we can refactor the parser to have two phases:</p>
<ul>
<li>parse-to-events: takes in tokens, produces some kind of event stream or other more "neutral" data structure</li>
<li>events-to-AST: something that constructs the AST from that?</li>
</ul>
<p>Is this correct?</p>
<p>EDIT: To elaborate, part of why I brought up recovery, is I wanted to understand whether recovery would like in parse-to-events or events-to-AST, I think probably the former...</p>



<a name="187650269"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650269" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650269">(Feb 07 2020 at 15:43)</a>:</h4>
<p>(Sorry, I was typing that while <span class="user-mention" data-user-id="126931">@centril</span> was asking their question, feel free to answer that first :)</p>



<a name="187650279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650279">(Feb 07 2020 at 15:43)</a>:</h4>
<p><span class="user-mention" data-user-id="126931">@centril</span> here's example where rustc doesn't even produce an ast: <a href="https://gist.github.com/matklad/5725e97192363c973c985e3d765d7fbf" target="_blank" title="https://gist.github.com/matklad/5725e97192363c973c985e3d765d7fbf">https://gist.github.com/matklad/5725e97192363c973c985e3d765d7fbf</a></p>



<a name="187650306"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650306" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650306">(Feb 07 2020 at 15:43)</a>:</h4>
<p>(not 100% sure about that, but <code>-Z ast-json</code> should show AST if rustc got to that phase?)</p>



<a name="187650323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650323">(Feb 07 2020 at 15:44)</a>:</h4>
<p>rust-analyzer produces the tree, and recognizes the <code>main</code> function</p>



<a name="187650367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650367">(Feb 07 2020 at 15:44)</a>:</h4>
<p>the example is</p>
<div class="codehilite"><pre><span></span><span class="k">use</span><span class="w"></span>

<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
</pre></div>



<a name="187650377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650377">(Feb 07 2020 at 15:44)</a>:</h4>
<p>And that is exactly the code the IDE sees when a user adds a new <code>use</code></p>



<a name="187650400"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650400" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650400">(Feb 07 2020 at 15:44)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> correct, yes; the current parser would return <code>Err(...)</code> and fail unless there's more elaborate recovery</p>



<a name="187650407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650407">(Feb 07 2020 at 15:45)</a>:</h4>
<p>what the current parser does a lot of is semantic recovery</p>



<a name="187650421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650421">(Feb 07 2020 at 15:45)</a>:</h4>
<p>e.g. <code>auto x = 0;</code> we will recognize</p>



<a name="187650443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650443">(Feb 07 2020 at 15:45)</a>:</h4>
<p>Yeah, and this is not the kind of recovery that is needed for ide use case. I suggest tabling recovery disucssion at this :)</p>



<a name="187650447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650447">(Feb 07 2020 at 15:45)</a>:</h4>
<p>sure</p>



<a name="187650519"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650519" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650519">(Feb 07 2020 at 15:46)</a>:</h4>
<blockquote>
<p>To elaborate, part of why I brought up recovery, is I wanted to understand whether recovery would like in parse-to-events or events-to-AST, I think probably the former...</p>
</blockquote>
<p>I think ide recovery belongs to the former, and semantic recovery kind of belongs to both</p>



<a name="187650539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650539">(Feb 07 2020 at 15:46)</a>:</h4>
<p>Does that imply my summary of the split is correct?</p>



<a name="187650550"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650550" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650550">(Feb 07 2020 at 15:46)</a>:</h4>
<p>Like, semantic recovery would insert explicit "missing expression" nodes into places where there isn't any code at all</p>



<a name="187650561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650561">(Feb 07 2020 at 15:47)</a>:</h4>
<p>Yes, I belive the summary is correct</p>



<a name="187650571"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650571" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650571">(Feb 07 2020 at 15:47)</a>:</h4>
<p>(Although I think I remember us discussing, specifically, whether we really want to have a "generate events" step, or whether we want a parser that can produce <em>both</em> AST and events, depdnding on configuration)</p>



<a name="187650591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650591">(Feb 07 2020 at 15:47)</a>:</h4>
<p>The main (only?) sticking point for me is:</p>
<blockquote>
<p>The tree is untyped. Each node has a “type tag”, SyntaxKind.</p>
<p>Syntax trees are simple value type. It is possible to create trees for a syntax without any external context.</p>
</blockquote>
<p>Making the parser untyped comes with all of the problems that untypedness entails. It's a great property of the current parser that I can simply make some change to the AST and then the compiler will yell at me regarding all the places that I need to change... and ofc typedness prevents bugs... So I'm not a fan of:</p>
<div class="codehilite"><pre><span></span><span class="c1">// test box_pat</span>
<span class="c1">// fn main() {</span>
<span class="c1">//     let box i = ();</span>
<span class="c1">//     let box Outer { box i, j: box Inner(box &amp;x) } = ();</span>
<span class="c1">//     let box ref mut i = ();</span>
<span class="c1">// }</span>
<span class="k">fn</span> <span class="nf">box_pat</span><span class="p">(</span><span class="n">p</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">Parser</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">CompletedMarker</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">assert</span><span class="o">!</span><span class="p">(</span><span class="n">p</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="n">T</span><span class="o">!</span><span class="p">[</span><span class="k">box</span><span class="p">]));</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">start</span><span class="p">();</span><span class="w"></span>
<span class="w">    </span><span class="n">p</span><span class="p">.</span><span class="n">bump</span><span class="p">(</span><span class="n">T</span><span class="o">!</span><span class="p">[</span><span class="k">box</span><span class="p">]);</span><span class="w"></span>
<span class="w">    </span><span class="n">pattern</span><span class="p">(</span><span class="n">p</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="n">m</span><span class="p">.</span><span class="n">complete</span><span class="p">(</span><span class="n">p</span><span class="p">,</span><span class="w"> </span><span class="n">BOX_PAT</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>



<a name="187650593"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650593" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650593">(Feb 07 2020 at 15:47)</a>:</h4>
<p>i want to add though, that I think an important part of "neutral"ity is being untagged</p>



<a name="187650680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650680">(Feb 07 2020 at 15:48)</a>:</h4>
<p>I would much prefer to e.g. have an interface based on associated types, which RA can dispatch to untyped things whereas rustc dispatches to typed things</p>



<a name="187650694"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650694" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650694">(Feb 07 2020 at 15:48)</a>:</h4>
<p>e.g. <code>type Pat;</code></p>



<a name="187650698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650698">(Feb 07 2020 at 15:48)</a>:</h4>
<p>and <code>type Expr;</code></p>



<a name="187650723"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650723" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650723">(Feb 07 2020 at 15:48)</a>:</h4>
<p>Yeah, this is the core  question!</p>



<a name="187650747"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650747" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650747">(Feb 07 2020 at 15:49)</a>:</h4>
<p>so what are the main benefits of being untagged?</p>



<a name="187650750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650750">(Feb 07 2020 at 15:49)</a>:</h4>
<p>I agree that typedness has benefits, but I also think that this is a tradeoff</p>



<a name="187650788"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650788" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650788">(Feb 07 2020 at 15:49)</a>:</h4>
<p><span class="user-mention" data-user-id="126931">@centril</span> can I make sure I understand your point? Are you saying that you like having the parser produce a "typed AST" (i.e., a set of enums that are tied very closely to rust's syntax) because it helps you ensure that the parser is "in sync". i.e., if I edit the AST to add a new field to some kind of expression, then I will get errors until I adjust the parser to supply values for that field?</p>



<a name="187650790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650790">(Feb 07 2020 at 15:49)</a>:</h4>
<blockquote>
<p>so what are the main benefits of being untagged?</p>
</blockquote>
<p>(or of <strong>supporting</strong> untaggedness)</p>



<a name="187650858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650858">(Feb 07 2020 at 15:50)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> <code>enum</code>s are what we have today</p>



<a name="187650886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650886">(Feb 07 2020 at 15:50)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> in terms of untagged <em>trees</em>, they are much better at representing incomplete trees and triva nodes</p>



<a name="187650896"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650896" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650896">(Feb 07 2020 at 15:50)</a>:</h4>
<p>what I'm suggesting is that we move from an initial (tagged) encoding with <code>enum</code>s to a final tagless encoding based on associated types</p>



<a name="187650902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650902">(Feb 07 2020 at 15:50)</a>:</h4>
<p>Typed trees by definition exclude certain impossible states, but with broken code any state is possible</p>



<a name="187650907"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650907" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650907">(Feb 07 2020 at 15:51)</a>:</h4>
<p>I can believe this; i.e. I can believe they handle the <code>use fn main() { }</code> case better.</p>



<a name="187650914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650914">(Feb 07 2020 at 15:51)</a>:</h4>
<p>In terms of untagged parser, it <strong>massively</strong> simplies the interface.</p>



<a name="187650917"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650917" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650917">(Feb 07 2020 at 15:51)</a>:</h4>
<p><span class="user-mention" data-user-id="126931">@centril</span> you didn't really answer my question though, did I understand you correctly as to the benefits?</p>



<a name="187650936"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650936" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650936">(Feb 07 2020 at 15:51)</a>:</h4>
<p>(I believe similar benefits could be obtained in other ways, as I think you are now suggesting)</p>



<a name="187650940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650940">(Feb 07 2020 at 15:51)</a>:</h4>
<p>Like, its a "trait with handful of methods" vs "trait with hundreds of associated types"</p>



<a name="187650953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187650953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187650953">(Feb 07 2020 at 15:51)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> yes, but also typedness generally prevents bugs. I cannot put something of the wrong type where it doesn't belong</p>



<a name="187651055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651055">(Feb 07 2020 at 15:52)</a>:</h4>
<p>It seems to me that there may be some interaction with pre-expansion gating, as well. In particular, if we have untyped trees, we might be able to represent things like "a function whose body is not yet parsed"</p>



<a name="187651065"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651065" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651065">(Feb 07 2020 at 15:52)</a>:</h4>
<blockquote>
<p>In terms of untagged parser, it <strong>massively</strong> simplies the interface.</p>
</blockquote>
<p>this is a distinct feature of untagged-ness, and unless I misunderstand, it is at direct odds with <span class="user-mention" data-user-id="126931">@centril</span>'s desire for static checking, right?</p>



<a name="187651069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651069">(Feb 07 2020 at 15:52)</a>:</h4>
<p>(and which is ultimately stripped before it becomes necessary to parse it)</p>



<a name="187651074"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651074" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651074">(Feb 07 2020 at 15:52)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> I think you are overestimating how many types there are in <a href="http://ast.rs" target="_blank" title="http://ast.rs">ast.rs</a></p>



<a name="187651096"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651096" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651096">(Feb 07 2020 at 15:53)</a>:</h4>
<blockquote>
<p>this is a distinct feature of untagged-ness, and unless I misunderstand, it is at direct odds with @centril's desire for static checking, right?</p>
</blockquote>
<p>Right</p>



<a name="187651113"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651113" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651113">(Feb 07 2020 at 15:53)</a>:</h4>
<p>(time check: 52 minutes; velocity check: I'm feeling a bit overwhelmed by how many threads of discussion are happening at once, apologies for adding to them :)</p>



<a name="187651138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651138">(Feb 07 2020 at 15:53)</a>:</h4>
<p>/me will try to capture some of what's been said in last few minutes into <a href="https://hackmd.io/MR4bSmcpSlOPhz6aSebpXQ" target="_blank" title="https://hackmd.io/MR4bSmcpSlOPhz6aSebpXQ">doc</a></p>



<a name="187651147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651147">(Feb 07 2020 at 15:53)</a>:</h4>
<p>There are <strong>246</strong> syntax kinds in rust-analyzer</p>



<a name="187651198"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651198" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651198">(Feb 07 2020 at 15:54)</a>:</h4>
<p>can you link to that list?</p>



<a name="187651229"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651229" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651229">(Feb 07 2020 at 15:54)</a>:</h4>
<p>rustc's ast is simpler, because it is an AST and not a CST, so it can elide details</p>



<a name="187651248"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651248" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651248">(Feb 07 2020 at 15:54)</a>:</h4>
<p>ugh... that's a whole heck of a lot more than in <a href="http://ast.rs" target="_blank" title="http://ast.rs">ast.rs</a></p>



<a name="187651274"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651274" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651274">(Feb 07 2020 at 15:54)</a>:</h4>
<p><a href="https://github.com/rust-analyzer/rust-analyzer/blob/1996762b1f2b9cb196cc879f0ce26d28a3c450c8/crates/ra_parser/src/syntax_kind/generated.rs#L12-L245" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/1996762b1f2b9cb196cc879f0ce26d28a3c450c8/crates/ra_parser/src/syntax_kind/generated.rs#L12-L245">https://github.com/rust-analyzer/rust-analyzer/blob/1996762b1f2b9cb196cc879f0ce26d28a3c450c8/crates/ra_parser/src/syntax_kind/generated.rs#L12-L245</a></p>



<a name="187651331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651331">(Feb 07 2020 at 15:55)</a>:</h4>
<p>specific to parser, the typedness also facilitates knowing what the actual grammar is</p>



<a name="187651333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651333">(Feb 07 2020 at 15:55)</a>:</h4>
<p>It also includes tokens, so I think we may save a 100 by not making type disticntions between tokens</p>



<a name="187651344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651344">(Feb 07 2020 at 15:55)</a>:</h4>
<p>but then we lose some type safety</p>



<a name="187651370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651370">(Feb 07 2020 at 15:55)</a>:</h4>
<p>I think the tradeoff from a rustc centric POV clearly speaks in favor of being typed</p>



<a name="187651443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651443">(Feb 07 2020 at 15:56)</a>:</h4>
<p>I can imagine there is a middle ground here, where we use associated types solely for the types already identified by <a href="http://ast.rs" target="_blank" title="http://ast.rs">ast.rs</a></p>



<a name="187651465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651465">(Feb 07 2020 at 15:56)</a>:</h4>
<p>For incomplete things we also have <code>ExprKind::Err</code> btw</p>



<a name="187651481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651481">(Feb 07 2020 at 15:56)</a>:</h4>
<p>I also don't think that typing <em>parsing</em> code is that important</p>



<a name="187651523"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651523" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651523">(Feb 07 2020 at 15:57)</a>:</h4>
<p>What is important, is having a fuzzable formal grammar, which you can check against the parser</p>



<a name="187651541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651541">(Feb 07 2020 at 15:57)</a>:</h4>
<p>Like, parser bugs are extremely easy and cheap to fix</p>



<a name="187651559"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651559" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651559">(Feb 07 2020 at 15:57)</a>:</h4>
<p>I don't agree, especially not when we have stability to contend with</p>



<a name="187651562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651562">(Feb 07 2020 at 15:58)</a>:</h4>
<p>well ... I don't know about that</p>



<a name="187651622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651622">(Feb 07 2020 at 15:58)</a>:</h4>
<p>(about being easy and cheap to fix, that is. especially when it comes to how recovery is implemented)</p>



<a name="187651656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651656">(Feb 07 2020 at 15:58)</a>:</h4>
<p>If RA accepts syntax it shouldn't, that doesn't have stability implications; for rustc it does</p>



<a name="187651661"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651661" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Esteban Küber <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651661">(Feb 07 2020 at 15:58)</a>:</h4>
<p>I guess the comment was meant for correct code, not recovered code</p>



<a name="187651695"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651695" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651695">(Feb 07 2020 at 15:59)</a>:</h4>
<p>I don't think I had a lot of parser bugs in rust-anlayzer, which could have be prevented by typeing.</p>
<p>Like, the today's bug is that we mixed up the order of <code>async</code> and <code>unsafe</code>.</p>



<a name="187651699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651699">(Feb 07 2020 at 15:59)</a>:</h4>
<blockquote>
<p>What is important, is having a fuzzable formal grammar, which you can check against the parser</p>
</blockquote>
<p>We also don't have this</p>



<a name="187651711"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651711" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651711">(Feb 07 2020 at 15:59)</a>:</h4>
<p>My gut feeling is that most parser bugs are like that</p>



<a name="187651807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651807">(Feb 07 2020 at 16:00)</a>:</h4>
<p>this is hard to answer definitively but it does feel like there may be a structure that gives some of the benefits of both</p>



<a name="187651823"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651823" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651823">(Feb 07 2020 at 16:00)</a>:</h4>
<p>that said, it's been one hour</p>



<a name="187651838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651838">(Feb 07 2020 at 16:00)</a>:</h4>
<p>yeah I gotta go</p>



<a name="187651846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651846">(Feb 07 2020 at 16:00)</a>:</h4>
<p>(despite being late in  the first place)</p>



<a name="187651854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651854">(Feb 07 2020 at 16:00)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> btw, did you consider the question re. <a href="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html#method.parse_delim_comma_seq" target="_blank" title="https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html#method.parse_delim_comma_seq">https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html#method.parse_delim_comma_seq</a> and whether we can do similar things in the new model?</p>



<a name="187651876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651876">(Feb 07 2020 at 16:00)</a>:</h4>
<p>these higher order combinators were important for recent cleanups to the parser</p>



<a name="187651890"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651890" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651890">(Feb 07 2020 at 16:01)</a>:</h4>
<p>things were quite messy before, and I would like not to get back into that situation</p>



<a name="187651892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651892">(Feb 07 2020 at 16:01)</a>:</h4>
<p>And not that we don't have parser bugs in rustc: <a href="https://github.com/rust-lang/rust/pull/37278" target="_blank" title="https://github.com/rust-lang/rust/pull/37278">https://github.com/rust-lang/rust/pull/37278</a></p>



<a name="187651927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651927">(Feb 07 2020 at 16:01)</a>:</h4>
<p><span class="user-mention" data-user-id="126931">@centril</span> high order combinations are combatible with event-based untagged parser I belive</p>



<a name="187651935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651935">(Feb 07 2020 at 16:01)</a>:</h4>
<p>I've done my best to <a href="https://hackmd.io/MR4bSmcpSlOPhz6aSebpXQ" target="_blank" title="https://hackmd.io/MR4bSmcpSlOPhz6aSebpXQ">make notes here</a>, I'd appreciate it if folks took a look and made changes/additions/corrections. I think it's great to keep discussing btw, but maybe try to include notes in that doc? I think I have to step away for a bit too.</p>



<a name="187651946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187651946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187651946">(Feb 07 2020 at 16:01)</a>:</h4>
<p>rust-analyzer's parsers started that way</p>



<a name="187652014"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652014" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652014">(Feb 07 2020 at 16:02)</a>:</h4>
<p>Thanks <span class="user-mention" data-user-id="116009">@nikomatsakis</span> !</p>



<a name="187652019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652019">(Feb 07 2020 at 16:02)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span>  yes sure, having fixed many bugs myself, I know that rustc also has parser bugs. They often come as a result of recovery, especially recovery encoded in ad-hoc ways</p>



<a name="187652025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652025">(Feb 07 2020 at 16:02)</a>:</h4>
<p>That said, before I go, I want to ask one question. I wrote this:</p>
<ul>
<li>Proposal is to split the "parsing function" into two steps:<ul>
<li>parse-to-events: takes in tokens, produces some kind of event stream or other more "neutral" data structure ("untagged trees")</li>
<li>events-to-AST: something that constructs the AST from that?</li>
</ul>
</li>
</ul>
<p>but I feel a bit of disconnect about this. Is "parse-to-events" accurate, or is it "parse-to-CST"?</p>



<a name="187652058"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652058" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652058">(Feb 07 2020 at 16:02)</a>:</h4>
<p>We can do both</p>



<a name="187652060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652060">(Feb 07 2020 at 16:02)</a>:</h4>
<p>those are not mutually exclusive?</p>



<a name="187652081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652081">(Feb 07 2020 at 16:03)</a>:</h4>
<p>parse-to-events is easier to agree upon, as events are simpler than CST</p>



<a name="187652111"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652111" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652111">(Feb 07 2020 at 16:03)</a>:</h4>
<p>I just feel like we didn't talk much about what these events would look like</p>



<a name="187652123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652123">(Feb 07 2020 at 16:03)</a>:</h4>
<p>But we can also decide that a specific CST format is reasonable, and parse directly to that</p>



<a name="187652135"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652135" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652135">(Feb 07 2020 at 16:03)</a>:</h4>
<p>But yes I agree not necessarily mutually exclusive. I'll follow-up later.</p>



<a name="187652185"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652185" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652185">(Feb 07 2020 at 16:04)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span>  btw, if you don't have to run, quick question... <span class="user-mention" data-user-id="119031">@Esteban Küber</span> often fancies using backtracking (e.g. cloning the parser state and then swapping it back) for better recovery</p>



<a name="187652189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652189">(Feb 07 2020 at 16:04)</a>:</h4>
<p>can RA deal with this?</p>



<a name="187652267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652267">(Feb 07 2020 at 16:05)</a>:</h4>
<p>It doesn't do backtracking, but I don't see why it can't be added</p>



<a name="187652307"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652307" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652307">(Feb 07 2020 at 16:05)</a>:</h4>
<p>Especially given that the parser state is relatively simpler</p>



<a name="187652393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652393">(Feb 07 2020 at 16:06)</a>:</h4>
<blockquote>
<p>I just feel like we didn't talk much about what these events would look like</p>
</blockquote>
<p>The interface rust-analyzer is using is literally this:</p>
<div class="codehilite"><pre><span></span><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">TreeSink</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">token</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kind</span>: <span class="nc">SyntaxKind</span><span class="p">,</span><span class="w"> </span><span class="n">n_tokens</span>: <span class="kt">u8</span><span class="p">);</span><span class="w"></span>

<span class="w">    </span><span class="k">fn</span> <span class="nf">start_node</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kind</span>: <span class="nc">SyntaxKind</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">finish_node</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">);</span><span class="w"></span>

<span class="w">    </span><span class="k">fn</span> <span class="nf">error</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">error</span>: <span class="nc">ParseError</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>



<a name="187652414"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652414" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652414">(Feb 07 2020 at 16:06)</a>:</h4>
<p>That is, for untyped events case</p>



<a name="187652803"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187652803" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187652803">(Feb 07 2020 at 16:10)</a>:</h4>
<p>btw, <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> wasn't here, but it would be good to hear from them before we make changes</p>



<a name="187653277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187653277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187653277">(Feb 07 2020 at 16:15)</a>:</h4>
<blockquote>
<p>But we can also decide that a specific CST format is reasonable</p>
</blockquote>
<p>This is actually an interesting point. I do think that the <code>Green</code> node layer from rust-analyzer is actually a pretty good CST in a sense that I don't think one can have a simpler interface, and that the impl of the interface does not seem outrageously ineffective.</p>
<p>So, if we are OK with</p>
<ul>
<li>CST being untagged</li>
<li>CST itself not storing hygiene information/global spans</li>
</ul>
<p>Than forgoing events and producing a green tree might be anintersting alternative</p>



<a name="187899966"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187899966" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187899966">(Feb 11 2020 at 10:33)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> btw; how are we going to handle editions if we don't have spans in the parser?</p>



<a name="187900456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187900456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187900456">(Feb 11 2020 at 10:39)</a>:</h4>
<p>Why do you need spans for that?  We don't handle editions in ra yet, but I think they could be handled nicely by passing an <code>edition</code> argument to the <code>parse</code> function, such that the parser can appropriately parse/reject edition-dependent constructs.</p>
<p>For edition lints, like "use <code>dyn Trait</code>", I think the same infra validation infra (post processing of CST) would work nicely.</p>



<a name="187901520"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187901520" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187901520">(Feb 11 2020 at 10:54)</a>:</h4>
<p>Would passing an edition argument work with macro calls between different editions work?</p>
<div class="codehilite"><pre><span></span><span class="c1">//edition 2015</span>
<span class="n">macro_rules</span><span class="o">!</span><span class="w"> </span><span class="n">my_macro</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="p">(</span><span class="o">@</span><span class="n">expr</span><span class="w"> </span><span class="cp">$a</span>:<span class="nc">expr</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cp">$a</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w">    </span><span class="p">(</span><span class="cp">$($a</span>:<span class="nc">tt</span><span class="p">)</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">();</span><span class="w"> </span><span class="n">my_macro</span><span class="o">!</span><span class="p">(</span><span class="o">@</span><span class="n">expr</span><span class="w"> </span><span class="cp">$($a</span><span class="p">)</span><span class="o">*</span><span class="p">);</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="c1">//edition 2018</span>
<span class="n">my_macro</span><span class="o">!</span><span class="p">(</span><span class="k">try</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="p">});</span><span class="w"></span>
</pre></div>


<p>should work I think, as the macro body is parsed as 2015 edition, while I think the <code>try { a }</code> is parsed as 2018 edition when passed to <code>my_macro!(@expr $($a)*));</code>.</p>



<a name="187901529"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187901529" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187901529">(Feb 11 2020 at 10:55)</a>:</h4>
<p>When deciding on what to parse when it comes to e.g. <code>.await</code> we check the span on the current token, e.g.:</p>
<div class="codehilite"><pre><span></span><span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">token</span><span class="p">.</span><span class="n">span</span><span class="p">.</span><span class="n">rust_2018</span><span class="p">()</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">eat_keyword</span><span class="p">(</span><span class="n">kw</span>::<span class="n">Await</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="k">return</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">mk_await_expr</span><span class="p">(</span><span class="n">self_arg</span><span class="p">,</span><span class="w"> </span><span class="n">lo</span><span class="p">);</span><span class="w"></span>
<span class="w">        </span><span class="p">}</span><span class="w"></span>
</pre></div>


<p>Changing this to a system based on "pass in the edition in <code>parse</code>" I am not sure how that would work out with edition hygiene (interactions with macros).<br>
I think you'll need to check with <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> to see if we can change systems here.</p>



<a name="187901555"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187901555" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187901555">(Feb 11 2020 at 10:55)</a>:</h4>
<blockquote>
<p>For edition lints, like "use dyn Trait", I think the same infra validation infra (post processing of CST) would work nicely.</p>
</blockquote>
<p>(We don't make parsing decisions wrt. <code>dyn Trait</code> based on the edition; that's done later.)</p>



<a name="187901632"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187901632" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187901632">(Feb 11 2020 at 10:56)</a>:</h4>
<p>Hmm... actually; come to think of it, doesn't the current setup of the rustc parser wrt. editions make it context sensitive? -- actually probably not...</p>



<a name="187901837"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187901837" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187901837">(Feb 11 2020 at 10:59)</a>:</h4>
<p>Wow, this does seem like an unfortunate interaction....</p>
<p>So, this is for cases where we pass a 2018 edition expression to 2015 edition macro?</p>
<p>Something like <code>log::info!("{}", my_future.await)</code>, where the intermediate result can also go via <code>$tt</code> phase...</p>
<p>That makes invisible delimiters model harder to make nice and self-contained :-(</p>



<a name="187901988"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187901988" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187901988">(Feb 11 2020 at 11:01)</a>:</h4>
<p>I don't recall exactly, but there are some tests I'm sure you could look at</p>



<a name="187902127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187902127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187902127">(Feb 11 2020 at 11:03)</a>:</h4>
<p>I guess this is surmountable: when the parser consumes an invisible delimiter, the TokenSource also informs the parser about the currently active edition. But it does add complexity to the interface.</p>



<a name="187902279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187902279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187902279">(Feb 11 2020 at 11:05)</a>:</h4>
<p>That might work. Seems preferable as a matter of specification as compared to spans + editions</p>



<a name="187902473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/design%20meeting%202020.02.07/near/187902473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/design.20meeting.202020.2E02.2E07.html#187902473">(Feb 11 2020 at 11:08)</a>:</h4>
<p>What about <code>macro_rules! m { ($a:tt) =&gt; { future.$a } }</code> in 2015 edition crate and then <code>m!(await)</code> in 2018 edition crate. I assume there is no invisible delimiter in that case.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>